Summary

PR improves the automatic and manual management of the amount of storage allocated by the Notifier (/api/system/tasks and /api/system/taskSummaries) API. In that context multiple new system settings were introduced to tune the automatic caps and cleanup behavior.

New System Settings

• notifierLogLevel: what level of messages to include in the log/list, default DEBUG (all)
• notifierMaxMessagesPerJob: each job can at most have this amount of messages in its list (soft enforced allowing momentary exceeding the limit by a few); default is 500
• notifierMaxAgeDays: job data older than this number of days is discarded (soft enforced, cleanup after 1 minute of idle); default is 7
• notifierMaxJobsPerType: if per JobType there are more than this number of jobs with data the oldest are discarded to get below this limit (soft enforced, cleanup after 1 minute idle); default is 500
• notifierGistOverview: when true the overview pages will only show the first and last message of the list for each job; default true
• notifierCleanAfterIdleTime: the time in milliseconds the notifier has to be idle (not moving messages from queue to store) before an automatic store cleanup is run using the notifierMaxAgeDays and notifierMaxJobsPerType as caps; default is 60sec

New API Endpoints

• DELETE /api/system/tasks: delete all data (of all jobs of any type)
• DELETE /api/system/tasks/{job-type}: delete all data of of all jobs of that type
• DELETE /api/system/tasks/{job-type}/{job-id}: delete all data of the job

When not just deleting data for a specific job (first 2 URLs) 2 optional parameters can be used:

• maxAge: limits the deletion to jobs that are at least of that age in days, e.g. maxAge=5 to delete all job data for jobs older than 5 days
• maxCount: limit the deletion so at most data for this number of jobs is kept, oldest jobs are deleted; e.g. maxCount=10 to keep data for the 10 jobs which have the most recent data

"All data" always means both notifications and summary. Therefore there is no equivalent deletion with /api/taskSummaries.

Asynchronous Notifications

The new implementation of the Notifier uses a BlockingQueue to decouple the sources of Notification messages from the storage part. This has several advantages, but mainly was needed to know that adding messages to the store is never concurrent to other writes or cleanup. That simplifies a lot how the operations can be implemented. As a bonus it should also improve performance of jobs slightly. In addition exceptions from notification and storage will never escalate to the job thread.

The single thread processing also allowed to make sure all messages will always have an increase of at least 1ms
in their timestamp and that all message timestamps reflect the actual store insert order. Timestamps are only adjusted when needed.

Storage Changes

• the JSON of a Notification itself got reduced since it contained multiple fields that can be recovered from context
• the summary is stored as JSON and only wrapped in JsonValue - it is never converted back to its object model (just to be turned into JSON again in the API layer) which has less overhead and less issues with changes, class renames, moves etcetera.
• in Redis only the notifications: key namespace is used to store the list of notifications (age of collection is extracted from most recent message)
• in Redis only the summaries: key namespace is used to store the summary as JSON (to remember the time the value is wrapped and unwrapped in another object, but the code can detect and handle existing values without the wrapper)
• A NotifierStore layer was introduced which handles the difference between in-memory and redis. The Notifier implementation itself is now common code. This allowed to keep each of the operations done in redis or in-memory so simple that it is easy to see that they are doing the right thing. This should help a lot in them both behaving the same.

Other Changes

• with this change all jobs will at least forward their first and last message to the Notifier (this is also to ensure that any summary added always has notifications - this simplifies the cleanup)
• the data value import job was still using the Notifier API directly in the service causing chaos with the messages where it would start and end twice and discard some of them. This was moved to the job and using JobProgress. A cleanup will be done in another PR.

Automatic Testing

Added new tests for the Notifier API including the added methods. The tests run with both InMemoryNotifierStore and RedisNotifierStore but using a FakeRedis. The FakeRedis itself is also tested directly to make sure it is a proper substitute for the real one.

Manual Testing

...